The recent advancements in small-size inference models facilitated AI deployment on the edge. However, the limited resource nature of edge devices poses new challenges especially for real-time applications. Deploying multiple inference models (or a single tunable model) varying in size and therefore accuracy and power consumption, in addition to an edge server inference model, can offer a dynamic system in which the allocation of inference models to inference jobs is performed according to the current resource conditions. Therefore, in this work, we tackle the problem of selectively allocating inference models to jobs or offloading them to the edge server to maximize inference accuracy under time and energy constraints. This problem is shown to be an instance of the unbounded multidimensional knapsack problem which is considered a strongly NP-hard problem. We propose a lightweight hybrid genetic algorithm (LGSTO) to solve this problem. We introduce a termination condition and neighborhood exploration techniques for faster evolution of populations. We compare LGSTO with the Naive and Dynamic programming solutions. In addition to classic genetic algorithms using different reproduction methods including NSGA-II, and finally we compare to other evolutionary methods such as Particle swarm optimization (PSO) and Ant colony optimization (ACO). Experiment results show that LGSTO performed 3 times faster than the fastest comparable schemes while producing schedules with higher average accuracy.

Machine learning models are usually developed in a training environment (online or offline) and then can be deployed to be used with live data. If you're working in Data Science and Machine learning projects, knowing how to deploy a model is one of the most important skills you'll need to have. Who is this article for? This article is for those who have created a machine learning model in a local machine and want to deploy and test the model within a short time. It's also for those who are looking for an alternative platform to deploy their machine learning models.

In this blog, guest writers Binyang Li (Software Engineer at Microsoft), Qianxi Zhang (Research Software Engineer at Microsoft), describe how to use Alluxio to solve the challenges while running inference at scale. The original content was published on Alluxio's Blog (Disclaimer: The author is a Founding Member @Alluxio). Offline inference, or batch inference, is an approach to run machine learning (ML) inference in a batch mode when processing a large dataset, as opposed to generating predictions in real-time given the input. The offline inference jobs are typically built on top of big data platforms to scale horizontally, and are running on fixed schedules (e.g. Running inference at scale is challenging.

The final push for the hat trick came down to the wire. Five minutes before the deadline, the team submitted work in its third and hardest data science competition of the year in recommendation systems. Called RecSys, it's a relatively new branch of computer science that's spawned one of the most widely used applications in machine learning, one that helps millions find what they want to watch, buy and play. The team's combination of six AI models packed into the contest's limit of 20 gigabytes all of the smarts it culled from studying 750 million data points. An unusual rule in the competition said the models had to run in less than 24 hours on a single core in a cloud CPU.

An open source machine learning library developed by researchers and engineers within Google's Machine Intelligence research organization. TensorFlow runs on multiple computers to distribute the training workloads. An open source framework built on top of TensorFlow that makes it easy to construct, train, and deploy object detection models. The Object Detection API provides pre-trained object detection models for users running inference jobs. Users are not required to train models from scratch.

Video analytics applications use edge compute servers for the analytics of the videos (for bandwidth and privacy). Compressed models that are deployed on the edge servers for inference suffer from data drift, where the live video data diverges from the training data. Continuous learning handles data drift by periodically retraining the models on new data. Our work addresses the challenge of jointly supporting inference and retraining tasks on edge servers, which requires navigating the fundamental tradeoff between the retrained model's accuracy and the inference accuracy. Our solution Ekya balances this tradeoff across multiple models and uses a micro-profiler to identify the models that will benefit the most by retraining. Ekya's accuracy gain compared to a baseline scheduler is 29% higher, and the baseline requires 4x more GPU resources to achieve the same accuracy as Ekya.